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The Research, Planning and Development Office of 
Area Technical College has developed a program evaluation 
that quantitatively assesses 13 variables considered to be 
indicators of program viability. These variables are used 
separate stages of the program evaluation: program 
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monitoring, in-depth program evaluation, and program 
reduction/discontinuance. Programs demonstrate their success in terms 
of their responses to 13 clearly defined criterion questions that can 
be answered with a simple yes or no answer. In addition each 
criterion is evaluated on a five point Likert Scale using specific 
quantitative standards. The evaluation procedure focuses on: (1) 
enrollment rate; (2) graduation rate; (3) graduate placement in 
appropriate field; (4) employer requests for graduates; (5) advisory 
committee recommendations; (6) cost per full-time equivalent student; 
(7) faculty productivity ratio; (8) relation to the college's 
mission; (9) needs assessment; (10) physical facilities; (11) 
equipment; (12) enrollment of female, minority, and handicapped 
students; and (13) availability of the same program from other 
agencies. Examples of how these criteria are applied are provided. 
(LAL) 
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QUANTITATIVE PROGRAM EVALUATION CRITERIA 



Multi-campus colleges and universities with diverse programs require certain 
agreed upon criteria for evaluating programs. Once these criteria are 
established, it is possible, in most cases, to develop quantitative measures to 
describe these criteria for each program. This paper is a presentation of just 
such a procedure developed and implemented at a large urban two-year technical 
college in the U.S.A. 

The Milwaukee Area Technical College has been evaluating programs and 
responding to changing job needs as a result of the change in the business and 
industrial mix in the industrial Midwest. The Research, Planning, and 
Development Office of the College developed a program evaluation procedure that 
quantitatively assessed thirteen variables that were considered by faculty, 
administrators, trustees, and business leaders to be important indicators of 
program viability. These variables are used in three separate stages of the 
program evaluation: 

1. Program Monitoring : All programs in the College are examined for 
enrollment rate, graduation rate, graduate placement, employer 
requests, cost per FTE, faculty productivity, relationship to the 
college mission, and the number of female, minority, and handicapped 
enrolled. This is to determine if there are any programs that require 
an in-depth evaluation or other special attention. 

2. In-depth Program Evaluation : Each year 20% of the programs are 
identified for an in-depth evaluation. This is done on a rotational 
basis insure that no in-depth evaluation is over five years old. 
This in-depth evaluation uses all of the program monitoring statistics 
plus advisory committee recommendation, a needs assessment, an 
examination of the physical facilities, equipment, and the availability 
of the program from other agencies. This and other information is 
presented to a program evaluation committee of faculty and cc liege 
administrators by the program faculty. The committee evaluates both 
the data and the faculty presentation and makes recommendations to be 
implemented in the College's strategic planning process. 

3. Program Reduction/Discontinuance : Although all thirteen criteria are 
used in the in-depth program evaluation process, only four criteria 
(weighted) are used to determine program reduction or discontinuance. 
These four and their weights are: five-year average enrollment (x4), 
five-year average graduation rate (x2), five-year average placement 
rate (x3), and long-term outlook (xl), 

PART I 

CRITERION REFERENCED EVALUATION 

The program evaluation committee endorsed the concept of criterion 
referenced evaluation to gi/e every program the opportunity to succeed 
by meeting thirteen clearly defined criterion questions that can be 



answered by a simple yes or no. These thirteen criteria and their 
questions are as follows: 

1. Enrollment rate : Is the enrollment rate at 80% or more of 
capacity? 

2. Graduation rate : Is the graduation rate 25% or greater? 

3. Graduate placement in appropriate field : Are 70% or more of che 
graduates placed in a related field? 

4. Employer requests : Do the number of job requests equal or exceed 
the number of graduates? 

5. A dvisory committee recommendation s: Has the advisory committee met 
during the last 12 months and recommended continuance of this 
program? 

6. Cost per FTE : Is the cost per FTE 200% of the College cost per FTE 
or less? 

7. Productivity ratio : Is the faculty productivity rario 50% of the 
College average or greater? 

8. Relation to the College mission : Is the program consistent with 
the College's current mission? 

9. Needs assessment : Is the curriculum meeting the current needs of 
employers as shown by our survey? 

10. Physical facilities : Are the physical facilities adequate for 
program objectives? 

11. Equipment : Is the equipment adequate for program objectives? 

12. Enrollment of female, minority, and handicapped : Does the program 
enroll at least 20% female, minority, or handicapped? 

13. Availability of program from other agencies : Will the elimination 
of the program seriously damage the supply of skilled workers in 
Metropolitan Milwaukee? 

Each program is placed in a category based upon the number of "yes" 
responses to the above questions. The categories are as follows: 

Category I: Acceptable, 12-13 yes responses 

II: Acceptable with some deficiencies, 10-11 yes responses 

III: Weak, 8-9 yes responses 

IV: Must be improved, 7 or less yes responses 
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PART II 

FIVE POINT LIKERT EVALUATION 



In addition to the simple "yes" and "no" questions, each criterion is evaluated 
on a five point likert scale using specific quantitative standards. These are 
shown as follows: 



1. Enrollment Rate (ratio of actual enrollment to slots available) 

Examine the enrollment rate over the last five years. This will establ"' ,h 
a trend. If the trend is positive, add 1 to the evaluation, if it is 
negative, subtract 1 from the evaluation. Rate the enrollment as follows: 



Enrollment Rate Rating 
80 - 100% 4 
60 - 79% 3 
40 - 59% 2 
0-39% 1 



Example : A program with a positive enrollment rate trend and an enrollment 
rate of 68% would receive 1 point for the positive rate and 3 points for 
being between 60 - 79%, for a total rating of 4. This program would still 
be checked as no because the enrollment rate was not 80% of capacity. 



2. Graduation Rate (the percent of students starting a program who actually 
graduate) 

Examine the graduation rate over the last 5 years. Rate as follows: 



Graduation Rate Rating 
80 - 100% 5 
60 - 79% 4 
40 - 59% 3 
20 - 39% 2 
0-19% 1 



Example : A program with a five-year average graduation rate of 29% would 
receive a rating of 2, but would be checked yes because the graduation rate 
was greater than 25%. 



3 . Graduate Placement in Appropriate Fie ld 

Examine the six-month student follow-up reports over the last 5 years to 
determine placement rate. Rate as follows: 
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Placement Rate Rating 
90 - 100% 5 
80 - 89% 4 
70 - 79% 3 
60 - 69% 2 
50 - 59% 1 



Example: A program with a five-year average placement rate of 85% would 
receive a rating of 4 and would be checked yes because more than 70% were 
placed in a related field. 



Employer Requests/Labor Market Information 

Examine the number of employee requests for graduates of the program and 

compare these requests to the number of students graduating. Rate as 
follows : 

Percent of Class Size Rating 
200% and over 5 
150 - 199% 4 
100 - 149% 3 
50 - 99% 2 
0-49% 1 



Example ; A program with 10 graduates and t J ob requests would have a job 
request rate of 60% and therefore a rating of 2 and would be checked n<o 
because this rate ts less than 99%. 



Advisory Committee Recommend tions 



The program advisory committee must meet within the 12 months prior to the 
program evaluation and make recommendations concerning the continuance of 
the program. Rate as follows: 



Advisory Committee Action Rating 

Advisory committee members employ graduates 

of the program and recommend it highly. 5 

Advisory committee members either employ 

graduates or recommend the program highly. 4 

Advisory committee members recommend a few 
improvements in the program. 3 

Advisory committee members recommend several 

improvements in the program to bring it up to 

their standard. 2 

Advisory committee members recommend discon- 
tinuance of the program. 1 

The advisory committee will vote YES or NO on the continuance of the 
program. 



Example : A program whose advisory committee recommended a few improvements 
and voted to continue the program would receive a rating of 3 and be 
checked yes . 



6 . Cost per FTE 

Examine the cost per FTE as shown in the most current NCHEMS report and 
compare this cost to the district-wide average for the same year. (Compare 
day programs to day averages and evening programs to evening averages and 
then average the ratings). Rate as follows: 



Percent of District-Wide Average Rating 

Loss than 100% 5 
101-150% 4 
151-200% 3 
201-250% 2 
Greater than 250% 1 



Example : A program with a cost per FTE of $3,542.20 will have the following 
percentage and rating: 

3,542.20 

2,323.56 = 152% Rating of 3 

The cost is less than 200% of the district average and is therefore checked 
as yes . 



7 . Productivity Ratio 

Examine the productivity ratio as shown in the most current NCHEMS report 
and compare to the district-wide average for the same year. Rate as 
follows : 



Percent of District-wide Average Rating 

Greater than 200% 5 
100-199% 4 
50- 99% 3 
25- 49% 2 
Less than 25% 1 



Example : A program with a productivity ratio of 404.28 when the 

district average is 368.34 will have the following percentage and rating: 

404.28 

368.34 = 110% Rating of 4 

The ratio is greater than 50% and is the efore checked as yes . 

8. Relation to MATC Mission 

Examine the program and place it under one of the five mission statements 
of our school. If it cannot be placed under one of the mission statements, 
check no on the evaluation worksheet. Evaluate as follows: 
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Mission 



Rating 



Vocational and technical education 
for entry into the job market. 



5 



Training and upgrading which is 
necessary for advancement or competence 
in a vocational and technical skill. 



4 



Remedial or special education which 
mav be a prerequisite to vocational 
training or employment . 3 

Courses that enable students to 
continue their education at other 

institutions . 2 

Opportunities for personal, civic, 

and cultural enrichment. 1 

(Note: These are the priorities established by the MATC Board in the 
Strategic Planning Statement.) 

Example : An Associate Degree program would be rated 5 because its mission 
is "Vocational-technical education for entry into the job market." It 
would be checked yes because it satisfies a mission of the school. 

9. Needs Assessment 

Examine the results of a survey of 15 to 25 employers recommended by the 
advisory committee. The survey will be constructed so that the results can 
be rated from 1 to 5. It will also have a category to check yes or ro 
relative to the needs assessment. If 80% of the respondents check "yes" 
then the yes will be checked on the worksheet. 

Example : A program witn a combined rating of 3.8 and 17 out of 20 
respondents checking yes will be given a rating of 4 (3.8 rounded) and will 
be checked yes (17 out of 25 is 85%). 



10. Physical Facilities 

Examine the physical facilities using the physical facilities worksheet. 
(This activity can be delegated to someone knowledgeable p^out the specific 
program needs). Rate as follows: 

The physical facilities worksheet will generate a combined rating from 
1 to 5 to describe the adequacy of the physical facilities. This 
combined rating is rounded to the nearest whole number and used as the 
rating on the program evaluation worksheet. This category will be 
checked yes if the physical facilities worksheet is checked yes. 

Example : A program with a physical facilities worksheet combined rating of 
3.4 and a yes check on the physical facilities worksheet will be given this 
rating: 3.4 is rounded to 3 and the program evaluation worksheet is 
checked yes because the physical facilities worksheet was checked yes. 
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11 . Equipment 

Examine the equipment using the equipment worksheet. (This activity can 
also be delegated to someone knowledgeable about the specific equipment 
needs of the program). Rate as follows: 

The equipment worksheet will yield from 1 to 5 checks in the yes 
column. The number of yes checks will be sued as the rating on the 
program evaluation worksheet. This category will be checked yes on 
the program evaluation worksheet if 3 of the 5 categories are checked 
yes on the equipment worksheet. 

Example : A program with 4 yes checks on the equipment worksheet will be 
given a rating of 4 and will be checked yes because more than 3 items are 
checked yes. 

12 . Impact on Target Population 

Examine current enrollment by sex and race to determine percentage in 
target population (female, minority, disadvantaged, handicapped). Evaluate 
as follows: 

Percentage of Target Populatio n Rating 



Example : A program with 26% enrollment from the target population would be 
rated 3 and be checked yes . 



13 . Availability of Programs from other Agencies 

Survey other schools and agencies in the district to determine if this 
program is available from other sources. Evaluate follows: 



40% or greater 



5 
4 
3 
2 
1 



30 - 39% 
20 - 29% 
10 - 19% 



less than 10% 



Rating 



No other programs offered. 



5 



Program offered that is 1/3 the 
size of our program. 



4 



Program offered totaling more 
than 1/3 the size of oit program 
but not greater than ours. 



3 



Program offered totaling more 
than our program. 



2 



Program offered totaling twice 
as many as our program. 



1 
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If the eli^ ion of the program will seriously damage the supply of 
skilJed workers in metropolitan Milwaukee, chock yes . 



Example : A program that graduates 12 per year and is similar to a program 

at MSOE graduating 15 per year would be rated 2. If both of these programs 
are needed to supply skilled workers, yes would be checked. 

These likert ratings assign a value representing the overall quality of the 

program. The ratings range from 1 to 5 and are evaluated as follows: 

Excellent 4.50 - 5.00 



PART III 

PROGRAM REDUCTION/DISCONTINUANCE CRITERIA 

Program reduction or discontinuance are determined by weighing three of the 
program evaluation criteria and determining a long-term outlook for a program 
based upon employer requests, projected new hires in two and five years, 
current labor market and other demographic information. This information has 
been gathered for forty-two of the programs and is presented in Table 1. A 
sample of the program reduction/discontinuance data for Chemical Technology is 
shown in Table 2. 



Very Poor 



Good 
Fair 
Poor 



3.75 - A. 49 
3.00 - 3.74 
2.50 - 2.99 
2.49 or less 
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TABLE 1 



Rank Order of Weighted Rating of Four Criteria 
(Enrollment, Graduation, Placement, Job Outlook) 



Rank Weigh Led 

Order Course Title Rating 



1 Jewelry Service and Sales - D (1986) 4.5 

2 Cosmetology - D 4.4 
3.5 *Nursing Assistant - C 4.3 
3.5 Child Care and Development - AD 4.3 
6.5 Chemical Technology - AD (1986) 4.2 
6.5 Dental Assistant - D (1986) 4.2 
6.5 Medical Assistant - D 4.2 
6.5 Respiratory Therapy - AD 4,2 

9 Marketing Management - AD (1986) 4.1 

10 Dental Hygiene - AD 4.0 

11 Radiography - AD (1986) 3.9 
12.5 Tool and Die Making - D 3.8 
12.5 Interior Design - AD 3.8 
15 Administrative Assistant - Secretarial - AD (1986) 3.6 
15 Electricity - D 3.6 
15 Food Preparation Assistant - D (1986) 3.6 
17.5 ^Horology - D 3.5 
17.5 Shoe Servicing - D (1986) 3.5 
20 *Alterationist/Tailoring - D 3.4 
20 Dental Laboratory Technology - AD 3.4 
20 Metallurgical Technology - AD 3.4 
22 Air Conditioning and Refrigeration Technology - AD 3.3 
23.5 ^Health Unit Clerk - C 3.2 
23.5 *Pharmacy Technical Aide - D 3.2 
26 *Air Conditioning, Refrigeration, and Heating - C 3.1 
26 Aboriculture/Urban Forestry - D 3.1 
26 ^Baking - D 3.1 

28 Electronics Servicing - D (1986) 3.0 

29 Combustion Engines Servicing - D 2.9 

30 ^Machine Molder/Loose Molder - C 2.8 
33.5 *Airf rame/Powerplant Mechanic - D 2.7 
33.5 Appliance Servicing - D (1986) 2.7 
33.5 Mechanical Drafting - D (1986) 2.7 
33.5 ^Welding Technology - AD 2.7 
33.5 *Structural Technician - AD 2.7 
33.5 *Architectural Technology - AD 2.7 
37.5 Combustion Engines Technology - AD (1986) 2.5 
37.5 *Public Works Technician - AD 2.5 
40 Environmental Health/Water & Wastewater Tech - AD (1986) 2.4 
40 *Hydraulics-Pneumat ics - D 2.4 
40 Occupational Music - AD 2.4 
42 ^Bricklaying and Masonry - D 2.1 

*Programs identified in Master Plan for possible reduction/discontinuance 



D - Diploma C - Certificate AD - Associate Degree 
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TABLE 2 

Reduction/Discontinuance Summary (1986) 

Program Number: (10)6-03:1 

Progiar Nar.c: Chemical Technology 

R ating Weight Weighted Rating 

5-Year Average Enrollment Capacity: 4 x 4 16 

5-Year Average Graduation Rate: 3 x 2 6 

5-Year Average Placement Rate: 5 3 15 

Long-Term Outlook: 5 x 1 5 

Total 42 
(T/10) Total Weighted Rating 4.2 
Average graduating class size: 9 
Long-Term Outlook was determined as follows: 

A. Are the number of employer requests at least equal to the graduating class 
size? yes 

B* Are the number of projected new-hires in two years greater than the average 
graduating class size? yes 

C. Are the number of projected new-hires in five years greater than the 
average graduating class size? yes 

D. Are the number of people currently employed 10 times greater than the 
average graduatii.g class size? (according to our survey) yes 

E. Any additional information. Survey indicates a continued demand for 
qualified graduates 
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